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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address « 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .1 36(a). tn no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- tf the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

• tf NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

• Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)^ Responsive to communication(s) filed on 11 August 2003 . 
2a)D This action is FINAL. 2b)K This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 7-42 is/are pending in the application. 

4a) Of the above claim(s) 24-26 and 36-42 is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) |S Claim(s) 1-23 and 27-35 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) S The specification is objected to by the Examiner. 

10) K The drawing(s) filed on 05 June 2000 is/are: a)M accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

11) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) 13 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)DAII b)D Some*c)l3 None of: 

1 .03 Certified copies of the priority documents have been received. 

2.D Certified copies of the priority documents have been received in Application No. . 



3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 

Attachment(s) 

1) (2 Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) CD Notice of Informal Patent Application (PTO-1 52) 

3) |3 Information Disclosure Statements) (PTO-1449) Paper No(s) 4. 6) □ Other: 
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DETAILED ACTION 
Election/Restrictions 

1. Applicant's election without traverse of group I (claims 1-23 and 27-35) in Paper No. 7 
filed August 1 1 , 2003 is acknowledged. 

Priority 

2. Acknowledgment is made of applicants claim for foreign priority based on applications 
filed in Australia on December 5, 1997 and April 8, 1998. 

Specification 

3. This application does not contain an abstract of the disclosure as required by 37 
CFR 1.72(b). An abstract on a separate sheet is required. 

Claim Objections 

4. Claim 3 is objected to under 37 CFR 1 .75(c) as being in improper form because a 
multiple dependent claim must refer to other claims only in the alternative. See MPEP 

§ 608.01(n). Accordingly, the claim has not been further treated on the merits. 

5. Claim 34 is objected to because of the following informalities: the last line of the claim, 
line 4, ends in a semicolon (";") and should end in a period ("."). Appropriate correction is 
required. 

Claim Rejections - 35 USC § 102 

6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in pubbc use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 
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7. Claims 1, 1 1 3 13, 15, 18, 27, 28, 34, and 35 are rejected under 35 U.S.C 102(b) as being 
anticipated by Geshwind et al. (USPN 4,925,294, newly cited), hereafter Geshwind. 

8. Regarding claim 1 , Geshwind discloses a method of producing a depth map for use in the 
conversion of 2D images into stereoscopic (3-D) images (column 3, line 2-16) including the 
steps of: identifying at least one object (image element) within a 2D image (column 3, line 23- 
28); allocating said or each object with an identifying tag (background 3 1 and 41; circles 32 and 
42; squares 33 and 43; Figure 1; column 3, line 17-28); allocating said or each object with a 
depth tag (depth position information; column 4, line 52-66); and determining and identifying an 
outline for said or each object (column 4, line 23-33, 44-49). 

9. Regarding claim 11, Geshwind discloses that the depth tag includes a color code (column 
7, line 12-27; column 8, line 1-5). 

10. Regarding claim 13, Geshwind discloses that said depth tag is a numerical value (depth, 
position information in the third dimension; column 4, line 52-55). 

11. Regarding claim 15, Geshwind discloses that said at least one object is further divided 
into a plurality of segments, each segment being assigned a depth tag (column 5, line 1-15). 

12. Regarding claim 18, Geshwind discloses tracking the or each object (image element) on 
successive frames of the image, and determining and assigning depth tags for the object in each 
respective frame (column 5, line 16-27). 

13. Regarding claim 27, Geshwind discloses a method of encoding a depth map for use in the 
conversion of 2D images into stereoscopic (3-D) images (column 3, line 2-16) including: 
allocating an object identifier to an object (background 3 1 and 41 ; circles 32 and 42; squares 33 
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and 43; Figure 1; column 3, line 17-28); allocating said object with a depth tag (depth position 
information; column 4, line 52-66); and defining the object outline (column 4, line 23-33, 44-49). 

14. Regarding claim 28, Geshwind discloses that said object outline is defined by a series of 
x,y coordinates (position information; column 4, line 23-31), each x,y coordinate being separated 
by a curve (inherent for a graphic input table and stylus provided to indicate the outline of the 
image elements). 

15. Regarding claim 34, Geshwind discloses that the encoding of the depth tag of said object 
includes: allocating a type of depth (uniform depth or different portions of single element given 
different depths; column 4, line 67-column 5, line 15); and allocating a depth for the object 
(column 4, line 52-55). 

16. Regarding claim 35, Geshwind discloses that the type of depth includes a single value 
(depth position information in the third dimension; column 4, line 52-55). It is noted that claim 
35 also claims that the type of depth includes linear ramp or radial ramp (emphasis added), since 
these elements are referred to in the alternative, any one element satisfies the claim, which is 
shown above. In addition, Geshwind discloses that a non-uniform depth may be assigned to 
different portions of a single image element (column 5, line 1-15) and it would be obvious to use 
a linear or radial ramp for this function (if these limitations were separately claimed). 

Claim Rejections - 35 USC §103 

17. The following is a quotation of 35 U.S.C 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this tide, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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18. Claims 2, 5, 6, 32, and 33 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Geshwind as applied to claims 1 and 27 above, and further in view of Miyakawa et al. (USPN 
4,783,829, newly cited), hereafter Miyakawa. 

19. Regarding claim 2, Geshwind discloses that the object outline is defined by a series of 
coordinates (position information), which obviously includes the drawing of curves to outline 
objects with the graphic input tablet and stylus (column 4, line 23-31), but does not explicitly 
disclose that the outline is defined by geometric shapes. Miyakawa discloses a pattern 
recognition apparatus wherein the outline of an object region is approximated by a polygon 
shape (column 3, line 53-62). It would have been obvious to one of ordinary skill in the art at the 
time the invention was made to define the outline by geometric shapes as taught by Miyakawa in 
order to allow pattern region information for a large amount of data to be highly compacted and 
the rotational pattern matching can be performed at high speed because the attitude rotation of 
the pattern is carried out by transforming the vertex coordinates of the polygon (column 2, line 
39-50). 

20. Regarding claim 5, Geshwind discloses that individual image elements are outlined by a 
human operator using a graphic input tablet and stylus, but does not explicitly disclose that 
determining the outline includes tracing the object pixel by pixel. The examiner takes Official 
Notice that determining the outline of an object by tracing the object pixel by pixel using a 
graphic input tablet and stylus, or a conventional computer mouse or other coordinate input 
device is well known in the art. It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to trace the object pixel by pixel as taught by Geshwind in order 
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to accurately determine the outline of the object so that it can be extracted or otherwise 
processed. 

21 . Regarding claim 6, Miyakawa discloses that the step of determining the outline further 
includes using straight (segmented) lines to approximate the outline of the object (column 4, line 
43-46; column 5, line 48-55). 

22. Regarding claim 32, Miyakawa discloses that said object outline is defined by at least one 
geometric shape (polygon approximation; see above discussion of claim 2). 

23. Regarding claim 33, Miyakawa discloses that said at least one geometric shape (polygon) 
is defined by the form of the shape (column 5, line 20-30) and the parameters (center of gravity, 
moment axis, etc.) of the shape (column 1 1, line 46-50). 

24. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over Geshwind in view 
of Miyakawa as applied to claims 1 and 2 above, and further in view of Eleftheriadis et al. 
(USPN 6,055,330, newly cited), hereafter Eleftheriadis. Regarding claim 3, Geshwind does not 
explicitly disclose that the identifying tag is a unique numerical number. Eleftheriadis discloses 
methods and apparatus for performing digital image and video segmentation and compression 
using 3-D depth information wherein each object mask is labeled with a different number which 
uniquely identifies an object (column 18, line 36-44). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to allocate each object with an 
identifying tag using a unique numerical number as taught by Eleftheriadis in order to uniquely 
identify the object (column 18, line 36-38). 
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25. Claims 4 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over Geshwind 
in view of Miyakawa as applied to claims 1 and 2 above, and further in view of Hsu (USPN 
5,640,468, newly cited). 

26. Regarding claim 4, Geshwind does not disclose that identifying said at least one object 
includes the step of comparing said 2D image with a library of generic scenes. Hsu discloses a 
method for identifying objects and features in an image including a set of self-determining and 
self-calibrating segmentation schemes and an image library comprising a mixture of model- 
based images and real-world images, known as a knowledge base or library (column 7, line 3-8) 
which are used to recognize objects by matching an observed (real world) image against a set of 
preset models (a library) which may be generate from a set of physical models or a set of wire 
frame models (column 15, line 50-67). It would have been obvious to one of ordinary skill in the 
art at the time the invention was made to identify said at least one object by comparing said 2D 
image with a library of generic scenes as taught by Hsu in order to provide a library construction 
scheme that is adaptable to both featureless and full/partial shape based object recognition 
scenarios as well as providing a dual library (depth and height) to perform general 3-D object 
recognition using any type of image (column 3, line 58-63). 

27. Regarding claim 9, Hsu discloses that the step of determining the outline further includes 
comparing the object with a library of curves (linear feature, edge-based pixels; column 14, line 
2-11) and/or generic or geometric shapes (column 6, line 45-51) to approximate the outline 
(column 7, line 3-8; column 15, line 50-67). 
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28. Claims 7 and 8 are rejected under 35 U.S.C. 103(a) as being unpatentable over Geshwind 
in view of Miyakawa as applied to claims 1 and 2 above, and further in view of Katayama et al. 
(USPN 6,404,936, newly cited), hereafter Katayama. 

29. Regarding claim 7, Geshwind does not disclose that the step of determining the outline 
further includes using curve approximations to approximate the outline of the object. Katayama 
discloses a subject image extraction method and apparatus wherein the surrounding portion of a 
contour is subjected to fitting using a smooth curve (column 8, line 12-19). It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to approximate the 
outline of an object using curve approximation as taught by Katayama in order to avoid 
cumbersome operations of manually drawing a smooth curve of a complicated shape using a 
mouse or the like (column 1, line 26-39). 

30. Regarding claim 8, Katayama discloses determining an outline using Bezier curves to 
approximate the outline of the object (column 8, line 12-19). 

31. Claim 10 is rejected under 35 U.S.C. 103(a) as being unpatentable over Geshwind in 
view of Miyakawa in view of Hsu as applied to claim 9 above, and further in view of Katayama. 

32. Regarding claim 10, Hsu does not explicitly disclose that the curve and/or generic or 
geometric shape are scaled to best fit the object. Katayama discloses that the surrounding 
portion of the contour is subjected to fitting using a smooth curve on the basis of the information 
of the contour generated by region growth (column 8, line 12-19), and if the generated contour 
has portions different from the subject contour, these portions are corrected by evaluating a 
degree of matching between the fitting curve and the subject contour and adding intermediate 
points to correct the curve fit (column 9, line 5-23). It would have been obvious to one of 
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ordinary skill in the art at the time the invention was made to scale the curve and/or generic or 
geometric shapes to best fit the object as taught by Katayama in order to avoid having to erase 
the contour and manually redraw a new contour as well as the cumbersome operations of 
manually drawing a smooth curve of a complicated shape using a mouse or the like (column 1 , 
line 26-39). 

33. Claim 12 is rejected under 35 U.S.C. 103(a) as being unpatentable over Geshwind as 
applied to claim 1 1 above, and further in view of Nourbakhsh et al. (USPN 5,793,900, cited on 
applicant's EDS), hereafter Nourbakhsh. 

34. Regarding claim 12, Geshwind does not disclose that white represents objects relatively 
close to the viewer, and black indicates objects relatively distant from the viewer. Nourbakhsh 
discloses generating categorical depth maps using passive defocus sensing wherein a depth map 
is an array of categorical depth values, each value indicating the depth of the scene for a given 
region such that depth values of 2, 1, and 0 correspond to close, medium, and far, respectively 
(column 5, line 9-15). Nourbakhsh further discloses that close regions are lightly shaded, 
medium regions are medium shaded, and far regions are darkly shaded (Figures 2-7; column 5, 
line 20-25). It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to represent objects relatively close to the viewer as white, and objects 
relatively distant from the viewer with black as taught by Nourbakhsh in order to give the viewer 
an impression of depth using varying pixel intensities since a brighter portion logically indicates 
a closer portion which is easier to see, and a darker portion indicates a distant portion which is 
more difficult to see (column 5, line 20-3 1). 
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35. Claims 14, 16, 17, and 23 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Geshwind as applied to claims 1,13, and 15 above. 

36. Regarding claim 14, Geshwind does not explicitly disclose that said depth tag is a 
numerical value that ranges from 0 to 255. The examiner takes Official Notice that encoding a 
numerical value that ranges from 0 to 255 or any other range for that matter is common and well 
known in the art of digital encoding schemes such as, in this case, an 8-bit binary number. It 
would have been obvious to one of ordinary skill in the art at the time the invention was made to 
define said depth tag as a numerical value that ranges from 0 to 255 in order to encode the depth 
tag as an 8-bit binary number. 

37. Regarding claims 16 and 17, Geshwind does not disclose that the variation in depth is 
defined by a linear or radial ramp function, but does disclose that different portions of a single 
image element may be given different depth designations with the computer interpolating depth 
coordinates over the entire element (column 5, line 1-15). The examiner takes Official Notice 
that using linear or radial ramp functions in order to introduce variations in an image is well 
known in the art. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to define the variation in depth by a linear or radial ramp function in order to 
create uniformly changing variations in the depth of a single image element. 

38. Regarding claim 23, Geshwind does not disclose producing greyscale images of 80x60x8 
bit resolution of each 2D image, but the specific dimensions of produced images depends on the 
design and requirements of the particular system. It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to produce grayscale images of 80x60x8 bit 
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resolution of each 2D image in order to satisfy specific design requirements of the particular 
system. 

39. Claims 19-22 are rejected under 35 U.S.C. 103(a) as being unpatentable over Geshwind 
as applied to claim 1 above, and further in view of Tseng et al. (WO 97/04404, cited on 
applicant's IDS), hereafter Tseng. 

40. Regarding claim 19, Geshwind discloses that random noise may be added to the depth 
information to eliminate the appearance of flat objects moving in space and to help achieve 
greater realism (column 5, line 28-30), but does not explicitly disclose adding a texture bump 
map to the or each object. Tseng discloses multi- viewpoint digital video encoding wherein a 
first matrix manipulator forms a mesh or 3-D matrix by combining the image and the depth map 
including a corresponding texture map incorporating the intensity values for each coordinate 
(page 8, line 26-page 9, line 5). It would have been obvious to one of ordinary skill in the art at 
the time the invention was made to add a texture bump map to the, or each, object as taught by 
Tseng in order to eliminate the appearance of flat objects moving in space and to help achieve 
greater realism (Geshwind, column 5, line 28-30). 

41 . Regarding claim 20, Geshwind discloses that said map is defined by breaking the object 
into a plurality of components and assigning each component a separate depth tag (column 5, 
line 1-15). 

42. Regarding claim 21 , Tseng discloses that said texture bump map is defined by the 
luminance values of individual components of the object (page 9, line 2-5). 
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43. Regarding claim 22, neither Geshwind nor Tseng disclose that a texture bump map is 
defined by the chrominance, saturation, color grouping, reflections, shadows, focus and/or 
sharpness of individual components of the object, but since Tseng discloses that said bump map 
is defined by the luminance of individual components of the object (see discussion of claim 21), 
it would have been obvious to one of ordinary skill in the art at the time the invention was made 
to define a texture bump map by the chrominance, saturation, color grouping, reflections, 
shadows, focus and/or sharpness of individual components of the object in order to eliminate the 
appearance of flat objects moving in space and to help achieve greater realism (Geshwind, 
column 5, line 28-30). 

44. Claims 29 and 31 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Geshwind as applied to claim 28 above, and further in view of Katayama. 

45. Regarding claim 29, Geshwind does not disclose that each said curve is stored in a library 
and allocated a unique number. Katayama discloses that the surrounding portion of a contour is 
subjected to fitting using a smooth curve on the basis of information of the contour generated by 
region growth (column 8, line 12-19). Katayama further discloses that an image is segmented 
before contour extraction, and contour extraction is done for an image in each segmented partial 
region (column 10, line 16-23), after which labels for identifying the segmented regions are 
given, and the labeled information is stored in a memory (column 10, line 58-64). Katayama 
does not explicitly disclose that each curve (segment) is allocated a unique number, but this is an 
obvious extension since each segmented region is labeled and stored which inherently include 
the contours extracted for each partial region. It would have been obvious to one of ordinary 
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skill in the art at the time the invention was made to allocate a unique number and store each 
curve in a library as taught by Katayama in order to segment an input image into a plurality of 
small regions and assign labels to the small regions in order to identify them (column 2, line 15- 
20), and to automatically correct contour points in units of labels such that no manual, 
complicated processing is required, and the contour can be easily corrected (column 2, line 35- 
40). 

46. Regarding claim 3 1 , Katayama discloses that each said curve is a Bezier curve (column 8, 
line 12-19). 

47. Claim 30 is rejected under 35 U.S.C. 103(a) as being unpatentable over Geshwind in 
view of Katayama as applied to claims 28 and 29 above, and further in view of Azarbayejani et 
al. (USPN 5,51 1,153, newly cited), hereafter Azarbayejani. 

48. Regarding claim 30, neither Geshwind nor Katayama disclose that said object outline 
also includes data on the orientation of each curve. Azarbayejani discloses a method and 
apparatus for three-dimensional, textured models from plural video images wherein an object is 
decomposed into piecewise smooth surfaces which are obtained by segmenting the original video 
into bounded two-dimensional regions, each of which can be represented using a single 
parameterized surface including spatial location, orientation, and scale of the surface (column 10, 
line 28-47). It is further disclosed that tracked feature points within polygons representing the 
various surface elements are used to compute vectors that define the spatial depth and orientation 
of a plane that includes the polygon (column 10, lien 57-65). It would have been obvious to one 
of ordinary skill in the art at the time the invention was made to include data on the orientation of 
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each curve in said object outline as taught by Azarbayejani in order to generate three- 
dimensional computer models from video without prior knowledge of camera geometry, thereby 
obviating the need for a calibrated camera (column 3, line 14-17). 

Conclusion 

49. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

• USPN 5,202,928 to Tomita et al. discloses a surface generation method from 
boundaries of stereo images. 

• USPN 5,109,425 to Lawton discloses a method and apparatus for predicting the 
direction of movement in machine vision including motion-parallax and depth maps. 

• USPN 5,710,875 to Harashima et al. discloses a method and apparatus for processing 
3-D multiple view images formed of a group of images obtained by viewing a 3-D 
object from a plurality of positions. 

• USPN 6,03 1,564 to Ma et al. discloses a method and apparatus for monoscopic to 
stereoscopic image conversion by outlining objects and adding depth information. 

• USPN 6, 1 8 1 ,8 1 5 to Marugame discloses a subject image extraction device including 
outlining and labeling. 

• USPN 6,504,569 to Jasinschi et al. discloses 2-D extended image generation from 3- 
D data extracted from a video sequence. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8:30 AM - 5 PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone number for the 
organization where this application or proceeding is assigned is (703) 872-9306. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-306-0377. 



rjh 

October 28, 2003 




